World's fastest brain-computer interface: Combining EEG2Code with deep learning.

We present a novel approach based on deep learning for decoding sensory information from non-invasively recorded Electroencephalograms (EEG). It can either be used in a passive Brain-Computer Interface (BCI) to predict properties of a visual stimulus the person is viewing, or it can be used to actively control a BCI application. Both scenarios were tested, whereby an average information transfer rate (ITR) of 701 bit/min was achieved for the passive BCI approach with the best subject achieving an online ITR of 1237 bit/min. Further, it allowed the discrimination of 500,000 different visual stimuli based on only 2 seconds of EEG data with an accuracy of up to 100%. When using the method for an asynchronous self-paced BCI for spelling, an average utility rate of 175 bit/min was achieved, which corresponds to an average of 35 error-free letters per minute. As the presented method extracts more than three times more information than the previously fastest approach, we suggest that EEG signals carry more information than generally assumed. Finally, we observed a ceiling effect such that information content in the EEG exceeds that required for BCI control, and therefore we discuss if BCI research has reached a point where the performance of non-invasive visual BCI control cannot be substantially improved anymore.

Asynchronous non-invasive high-speed BCI speller with robust non-control state detection.

Brain-Computer Interfaces (BCIs) enable users to control a computer by using pure brain activity. Recent BCIs based on visual evoked potentials (VEPs) have shown to be suitable for high-speed communication. However, all recent high-speed BCIs are synchronous, which means that the system works with fixed time slots so that the user is not able to select a command at his own convenience, which poses a problem in real-world applications. In this paper, we present the first asynchronous high-speed BCI with robust distinction between intentional control (IC) and non-control (NC), with a nearly perfect NC state detection of only 0.075 erroneous classifications per minute. The resulting asynchronous speller achieved an average information transfer rate (ITR) of 122.7 bit/min using a 32 target matrix-keyboard. Since the method is based on random stimulation patterns it allows to use an arbitrary number of targets for any application purpose, which was shown by using an 55 target German QWERTZ-keyboard layout which allowed the participants to write an average of 16.1 (up to 30.7) correct case-sensitive letters per minute. As the presented system is the first asynchronous high-speed BCI speller with a robust non-control state detection, it is an important step for moving BCI applications out of the lab and into real-life.

Fully Automated R-peak Detection Algorithm (FLORA) for fetal magnetoencephalographic data.

BACKGROUND AND OBJECTIVE: Fetal magnetoencephalography (fMEG) is a method for recording fetal brain signals, fetal and maternal heart activity simultaneously. The identification of the R-peaks of the heartbeats forms the basis for later heart rate (HR) and heart rate variability (HRV) analysis. The current procedure for the evaluation of fetal magnetocardiograms (fMCG) is either semi-automated evaluation using template matching (SATM) or Hilbert transformation algorithm (HTA). However, none of the methods available at present works reliable for all datasets. METHODS: Our aim was to develop a unitary, responsive and fully automated R-peak detection algorithm (FLORA) that combines and enhances both of the methods used up to now. RESULTS: The evaluation of all methods on 55 datasets verifies that FLORA outperforms both of these methods as well as a combination of the two, which applies in particular to data of fetuses at earlier gestational age. CONCLUSION: The combined analysis shows that FLORA is capable of providing good, stable and reproducible results without manual intervention.

Fully Automated Subtraction of Heart Activity for Fetal Magnetoencephalography Data.

Fetal magnetoencephalography (fMEG) is a method to record human fetal brain signals in pregnant mothers. Nevertheless the amplitude of the fetal brain signal is very small and the fetal brain signal is overlaid by interfering signals mainly caused by maternal and fetal heart activity. Several methods are used to attenuate the interfering signals for the extraction of the fetal brain signal. However currently used methods are often affected by a reduction of the fetal brain signal or redistribution of the fetal brain signal. To overcome this limitation we developed a new fully automated procedure for removal of heart activity (FAUNA) based on Principal Component Analysis (PCA) and Ridge Regression. We compared the results with an orthogonal projection (OP) algorithm which is widely used in fetal research. The analysis was performed on simulated data sets containing spontaneous and averaged brain activity. The new analysis was able to extract fetal brain signals with an increased signal to noise ratio and without redistribution of activity across sensors compared to OP. The attenuation of interfering heart signals in fMEG data was significantly improved by FAUNA and supports fully automated evaluation of fetal brain signal.

On the design of EEG-based movement decoders for completely paralyzed stroke patients.

BACKGROUND: Brain machine interface (BMI) technology has demonstrated its efficacy for rehabilitation of paralyzed chronic stroke patients. The critical component in BMI-training consists of the associative connection (contingency) between the intention and the feedback provided. However, the relationship between the BMI design and its performance in stroke patients is still an open question. METHODS: In this study we compare different methodologies to design a BMI for rehabilitation and evaluate their effects on movement intention decoding performance. We analyze the data of 37 chronic stroke patients who underwent 4 weeks of BMI intervention with different types of association between their brain activity and the proprioceptive feedback. We simulate the pseudo-online performance that a BMI would have under different conditions, varying: (1) the cortical source of activity (i.e., ipsilesional, contralesional, bihemispheric), (2) the type of spatial filter applied, (3) the EEG frequency band, (4) the type of classifier; and also evaluated the use of residual EMG activity to decode the movement intentions. RESULTS: We observed a significant influence of the different BMI designs on the obtained performances. Our results revealed that using bihemispheric beta activity with a common average reference and an adaptive support vector machine led to the best classification results. Furthermore, the decoding results based on brain activity were significantly higher than those based on muscle activity. CONCLUSIONS: This paper underscores the relevance of the different parameters used to decode movement, using EEG in severely paralyzed stroke patients. We demonstrated significant differences in performance for the different designs, which supports further research that should elucidate if those approaches leading to higher accuracies also induce higher motor recovery in paralyzed stroke patients.

Design and effectiveness evaluation of mirror myoelectric interfaces: a novel method to restore movement in hemiplegic patients.

The motor impairment occurring after a stroke is characterized by pathological muscle activation patterns or synergies. However, while robot-aided myoelectric interfaces have been proposed for stroke rehabilitation, they do not address this issue, which might result in inefficient interventions. Here, we present a novel paradigm that relies on the correction of the pathological muscle activity as a way to elicit rehabilitation, even in patients with complete paralysis. Previous studies demonstrated that there are no substantial inter-limb differences in the muscle synergy organization of healthy individuals. We propose building a subject-specific model of muscle activity from the healthy limb and mirroring it to use it as a learning tool for the patient to reproduce the same healthy myoelectric patterns on the paretic limb during functional task training. Here, we aim at understanding how this myoelectric model, which translates muscle activity into continuous movements of a 7-degree of freedom upper limb exoskeleton, could transfer between sessions, arms and tasks. The experiments with 8 healthy individuals and 2 chronic stroke patients proved the feasibility and effectiveness of such myoelectric interface. We anticipate the proposed method to become an efficient strategy for the correction of maladaptive muscle activity and the rehabilitation of stroke patients.

Unity and diversity in working memory load: Evidence for the separability of the executive functions updating and inhibition using machine learning.

OBJECTIVE: According to current theoretical models of working memory (WM), executive functions (EFs) like updating, inhibition and shifting play an important role in WM functioning. The models state that EFs highly correlate with each other but also have some individual variance which makes them separable processes. Since this theory has mostly been substantiated with behavioral data like reaction time and the ability to execute a task correctly, the aim of this paper is to find evidence for diversity (unique properties) of the EFs updating and inhibition in neural correlates of EEG data by means of using brain-computer interface (BCI) methods as a research tool. To highlight the benefit of this approach we compare this new methodology to classical analysis approaches. METHODS: An existing study has been reinvestigated by applying neurophysiological analysis in combination with support vector machine (SVM) classification on recorded electroencephalography (EEG) data to determine the separability and variety of the two EFs updating and inhibition on a single trial basis. RESULTS: The SVM weights reveal a set of distinct features as well as a set of shared features for the two EFs updating and inhibition in the theta and the alpha band power. SIGNIFICANCE: In this paper we find evidence that correlates for unity and diversity of EFs can be found in neurophysiological data. Machine learning approaches reveal shared but also distinct properties for the EFs. This study shows that using methods from brain-computer interface (BCI) research, like machine learning, as a tool for the validation of psychological models and theoretical constructs is a new approach that is highly versatile and could lead to many new insights.

Modelling the brain response to arbitrary visual stimulation patterns for a flexible high-speed Brain-Computer Interface.

Visual evoked potentials (VEPs) can be measured in the EEG as response to a visual stimulus. Commonly, VEPs are displayed by averaging multiple responses to a certain stimulus or a classifier is trained to identify the response to a certain stimulus. While the traditional approach is limited to a set of predefined stimulation patterns, we present a method that models the general process of VEP generation and thereby can be used to predict arbitrary visual stimulation patterns from EEG and predict how the brain responds to arbitrary stimulation patterns. We demonstrate how this method can be used to model single-flash VEPs, steady state VEPs (SSVEPs) or VEPs to complex stimulation patterns. It is further shown that this method can also be used for a high-speed BCI in an online scenario where it achieved an average information transfer rate (ITR) of 108.1 bit/min. Furthermore, in an offline analysis, we show the flexibility of the method allowing to modulate a virtually unlimited amount of targets with any desired trial duration resulting in a theoretically possible ITR of more than 470 bit/min.

The effect of monitor raster latency on VEPs, ERPs and Brain-Computer Interface performance.

BACKGROUND: Visual neuroscience experiments and Brain-Computer Interface (BCI) control often require strict timings in a millisecond scale. As most experiments are performed using a personal computer (PC), the latencies that are introduced by the setup should be taken into account and be corrected. As a standard computer monitor uses a rastering to update each line of the image sequentially, this causes a monitor raster latency which depends on the position, on the monitor and the refresh rate. NEW METHOD: We technically measured the raster latencies of different monitors and present the effects on visual evoked potentials (VEPs) and event-related potentials (ERPs). Additionally we present a method for correcting the monitor raster latency and analyzed the performance difference of a code-modulated VEP BCI speller by correcting the latency. COMPARISON WITH EXISTING METHODS: There are currently no other methods validating the effects of monitor raster latency on VEPs and ERPs. RESULTS: The timings of VEPs and ERPs are directly affected by the raster latency. Furthermore, correcting the raster latency resulted in a significant reduction of the target prediction error from 7.98% to 4.61% and also in a more reliable classification of targets by significantly increasing the distance between the most probable and the second most probable target by 18.23%. CONCLUSIONS: The monitor raster latency affects the timings of VEPs and ERPs, and correcting resulted in a significant error reduction of 42.23%. It is recommend to correct the raster latency for an increased BCI performance and methodical correctness.

Online EEG-Based Workload Adaptation of an Arithmetic Learning Environment.

In this paper, we demonstrate a closed-loop EEG-based learning environment, that adapts instructional learning material online, to improve learning success in students during arithmetic learning. The amount of cognitive workload during learning is crucial for successful learning and should be held in the optimal range for each learner. Based on EEG data from 10 subjects, we created a prediction model that estimates the learner's workload to obtain an unobtrusive workload measure. Furthermore, we developed an interactive learning environment that uses the prediction model to estimate the learner's workload online based on the EEG data and adapt the difficulty of the learning material to keep the learner's workload in an optimal range. The EEG-based learning environment was used by 13 subjects to learn arithmetic addition in the octal number system, leading to a significant learning effect. The results suggest that it is feasible to use EEG as an unobtrusive measure of cognitive workload to adapt the learning content. Further it demonstrates that a promptly workload prediction is possible using a generalized prediction model without the need for a user-specific calibration.

Classification of different reaching movements from the same limb using EEG.

OBJECTIVE: Brain-computer-interfaces (BCIs) have been proposed not only as assistive technologies but also as rehabilitation tools for lost functions. However, due to the stochastic nature, poor spatial resolution and signal to noise ratio from electroencephalography (EEG), multidimensional decoding has been the main obstacle to implement non-invasive BCIs in real-live rehabilitation scenarios. This study explores the classification of several functional reaching movements from the same limb using EEG oscillations in order to create a more versatile BCI for rehabilitation. APPROACH: Nine healthy participants performed four 3D center-out reaching tasks in four different sessions while wearing a passive robotic exoskeleton at their right upper limb. Kinematics data were acquired from the robotic exoskeleton. Multiclass extensions of Filter Bank Common Spatial Patterns (FBCSP) and a linear discriminant analysis (LDA) classifier were used to classify the EEG activity into four forward reaching movements (from a starting position towards four target positions), a backward movement (from any of the targets to the starting position and rest). Recalibrating the classifier using data from previous or the same session was also investigated and compared. MAIN RESULTS: Average EEG decoding accuracy were significantly above chance with 67%, 62.75%, and 50.3% when decoding three, four and six tasks from the same limb, respectively. Furthermore, classification accuracy could be increased when using data from the beginning of each session as training data to recalibrate the classifier. SIGNIFICANCE: Our results demonstrate that classification from several functional movements performed by the same limb is possible with acceptable accuracy using EEG oscillations, especially if data from the same session are used to recalibrate the classifier. Therefore, an ecologically valid decoding could be used to control assistive or rehabilitation mutli-degrees of freedom (DoF) robotic devices using EEG data. These results have important implications towards assistive and rehabilitative neuroprostheses control in paralyzed patients.

A high-speed brain-computer interface (BCI) using dry EEG electrodes.

Recently, brain-computer interfaces (BCIs) based on visual evoked potentials (VEPs) have been shown to achieve remarkable communication speeds. As they use electroencephalography (EEG) as non-invasive method for recording neural signals, the application of gel-based EEG is time-consuming and cumbersome. In order to achieve a more user-friendly system, this work explores the usability of dry EEG electrodes with a VEP-based BCI. While the results show a high variability between subjects, they also show that communication speeds of more than 100 bit/min are possible using dry EEG electrodes. To reduce performance variability and deal with the lower signal-to-noise ratio of the dry EEG electrodes, an averaging method and a dynamic stopping method were introduced to the BCI system. Those changes were shown to improve performance significantly, leading to an average classification accuracy of 76% with an average communication speed of 46 bit/min, which is equivalent to a writing speed of 8.8 error-free letters per minute. Although the BCI system works substantially better with gel-based EEG, dry EEG electrodes are more user-friendly and still allow high-speed BCI communication.

Electroencephalography Based Analysis of Working Memory Load and Affective Valence in an N-back Task with Emotional Stimuli.

Most brain-based measures of the electroencephalogram (EEG) are used in highly controlled lab environments and only focus on narrow mental states (e.g., working memory load). However, we assume that outside the lab complex multidimensional mental states are evoked. This could potentially create interference between EEG signatures used for identification of specific mental states. In this study, we aimed to investigate more realistic conditions and therefore induced a combination of working memory load and affective valence to reveal potential interferences in EEG measures. To induce changes in working memory load and affective valence, we used a paradigm which combines an N-back task (for working memory load manipulation) with a standard method to induce affect (affective pictures taken from the International Affective Picture System (IAPS) database). Subjective ratings showed that the experimental task was successful in inducing working memory load as well as affective valence. Additionally, performance measures were analyzed and it was found that behavioral performance decreased with increasing workload as well as negative valence, showing that affective valence can have an effect on cognitive processing. These findings are supported by changes in frontal theta and parietal alpha power, parameters used for measuring of working memory load in the EEG. However, these EEG measures are influenced by the negative valence condition as well and thereby show that detection of working memory load is sensitive to affective contexts. Unexpectedly, we did not find any effects for EEG measures typically used for affective valence detection (Frontal Alpha Asymmetry (FAA)). Therefore we assume that the FAA measure might not be usable if cognitive workload is induced simultaneously. We conclude that future studies should account for potential context-specifity of EEG measures.

Hybrid Neuroprosthesis for the Upper Limb: Combining Brain-Controlled Neuromuscular Stimulation with a Multi-Joint Arm Exoskeleton.

Brain-machine interface-controlled (BMI) neurofeedback training aims to modulate cortical physiology and is applied during neurorehabilitation to increase the responsiveness of the brain to subsequent physiotherapy. In a parallel line of research, robotic exoskeletons are used in goal-oriented rehabilitation exercises for patients with severe motor impairment to extend their range of motion (ROM) and the intensity of training. Furthermore, neuromuscular electrical stimulation (NMES) is applied in neurologically impaired patients to restore muscle strength by closing the sensorimotor loop. In this proof-of-principle study, we explored an integrated approach for providing assistance as needed to amplify the task-related ROM and the movement-related brain modulation during rehabilitation exercises of severely impaired patients. For this purpose, we combined these three approaches (BMI, NMES, and exoskeleton) in an integrated neuroprosthesis and studied the feasibility of this device in seven severely affected chronic stroke patients who performed wrist flexion and extension exercises while receiving feedback via a virtual environment. They were assisted by a gravity-compensating, seven degree-of-freedom exoskeleton which was attached to the paretic arm. NMES was applied to the wrist extensor and flexor muscles during the exercises and was controlled by a hybrid BMI based on both sensorimotor cortical desynchronization (ERD) and electromyography (EMG) activity. The stimulation intensity was individualized for each targeted muscle and remained subthreshold, i.e., induced no overt support. The hybrid BMI controlled the stimulation significantly better than the offline analyzed ERD (p = 0.028) or EMG (p = 0.021) modality alone. Neuromuscular stimulation could be well integrated into the exoskeleton-based training and amplified both the task-related ROM (p = 0.009) and the movement-related brain modulation (p = 0.019). Combining a hybrid BMI with neuromuscular stimulation and antigravity assistance augments upper limb function and brain activity during rehabilitation exercises and may thus provide a novel restorative framework for severely affected stroke patients.

EEG Responses to Auditory Stimuli for Automatic Affect Recognition.

Brain state classification for communication and control has been well established in the area of brain-computer interfaces over the last decades. Recently, the passive and automatic extraction of additional information regarding the psychological state of users from neurophysiological signals has gained increased attention in the interdisciplinary field of affective computing. We investigated how well specific emotional reactions, induced by auditory stimuli, can be detected in EEG recordings. We introduce an auditory emotion induction paradigm based on the International Affective Digitized Sounds 2nd Edition (IADS-2) database also suitable for disabled individuals. Stimuli are grouped in three valence categories: unpleasant, neutral, and pleasant. Significant differences in time domain domain event-related potentials are found in the electroencephalogram (EEG) between unpleasant and neutral, as well as pleasant and neutral conditions over midline electrodes. Time domain data were classified in three binary classification problems using a linear support vector machine (SVM) classifier. We discuss three classification performance measures in the context of affective computing and outline some strategies for conducting and reporting affect classification studies.

Asynchronous P300 classification in a reactive brain-computer interface during an outlier detection task.

OBJECTIVE: In this study, the feasibility of detecting a P300 via an asynchronous classification mode in a reactive EEG-based brain-computer interface (BCI) was evaluated. The P300 is one of the most popular BCI control signals and therefore used in many applications, mostly for active communication purposes (e.g. P300 speller). As the majority of all systems work with a stimulus-locked mode of classification (synchronous), the field of applications is limited. A new approach needs to be applied in a setting in which a stimulus-locked classification cannot be used due to the fact that the presented stimuli cannot be controlled or predicted by the system. APPROACH: A continuous observation task requiring the detection of outliers was implemented to test such an approach. The study was divided into an offline and an online part. MAIN RESULTS: Both parts of the study revealed that an asynchronous detection of the P300 can successfully be used to detect single events with high specificity. It also revealed that no significant difference in performance was found between the synchronous and the asynchronous approach. SIGNIFICANCE: The results encourage the use of an asynchronous classification approach in suitable applications without a potential loss in performance.

Error-related potentials during continuous feedback: using EEG to detect errors of different type and severity.

When a person recognizes an error during a task, an error-related potential (ErrP) can be measured as response. It has been shown that ErrPs can be automatically detected in tasks with time-discrete feedback, which is widely applied in the field of Brain-Computer Interfaces (BCIs) for error correction or adaptation. However, there are only a few studies that concentrate on ErrPs during continuous feedback. With this study, we wanted to answer three different questions: (i) Can ErrPs be measured in electroencephalography (EEG) recordings during a task with continuous cursor control? (ii) Can ErrPs be classified using machine learning methods and is it possible to discriminate errors of different origins? (iii) Can we use EEG to detect the severity of an error? To answer these questions, we recorded EEG data from 10 subjects during a video game task and investigated two different types of error (execution error, due to inaccurate feedback; outcome error, due to not achieving the goal of an action). We analyzed the recorded data to show that during the same task, different kinds of error produce different ErrP waveforms and have a different spectral response. This allows us to detect and discriminate errors of different origin in an event-locked manner. By utilizing the error-related spectral response, we show that also a continuous, asynchronous detection of errors is possible. Although the detection of error severity based on EEG was one goal of this study, we did not find any significant influence of the severity on the EEG.

Comparing metrics to evaluate performance of regression methods for decoding of neural signals.

The use of regression methods for decoding of neural signals has become popular, with its main applications in the field of Brain-Machine Interfaces (BMIs) for control of prosthetic devices or in the area of Brain-Computer Interfaces (BCIs) for cursor control. When new methods for decoding are being developed or the parameters for existing methods should be optimized to increase performance, a metric is needed that gives an accurate estimate of the prediction error. In this paper, we evaluate different performance metrics regarding their robustness for assessing prediction errors. Using simulated data, we show that different kinds of prediction error (noise, scaling error, bias) have different effects on the different metrics and evaluate which methods are best to assess the overall prediction error, as well as the individual types of error. Based on the obtained results we can conclude that the most commonly used metrics correlation coefficient (CC) and normalized root-mean-squared error (NRMSE) are well suited for evaluation of cross-validated results, but should not be used as sole criterion for cross-subject or cross-session evaluations.

A Brain-Computer Interface (BCI) system to use arbitrary Windows applications by directly controlling mouse and keyboard.

A Brain-Computer Interface (BCI) allows to control a computer by brain activity only, without the need for muscle control. In this paper, we present an EEG-based BCI system based on code-modulated visual evoked potentials (c-VEPs) that enables the user to work with arbitrary Windows applications. Other BCI systems, like the P300 speller or BCI-based browsers, allow control of one dedicated application designed for use with a BCI. In contrast, the system presented in this paper does not consist of one dedicated application, but enables the user to control mouse cursor and keyboard input on the level of the operating system, thereby making it possible to use arbitrary applications. As the c-VEP BCI method was shown to enable very fast communication speeds (writing more than 20 error-free characters per minute), the presented system is the next step in replacing the traditional mouse and keyboard and enabling complete brain-based control of a computer.